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ABSTRACT 



The structure of an organization plays a vital role 
in the evaluation of the organization. Social science researchers 
often assume that controls inherent in the physical sciences are as 
applicable to human subjects- Evaluation of Head Start is an example 
of the social relation of investigator to subject matter and of the 
variables introduced by the organizational structure of a complex 
research enterprise. (MLF) 
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Management Structures and Large-Scale Studies 



We will discuss today an aspect of large-scale research which has 
received little attention from the educational researcher, the organ" 
ization and management of the action project under study. This problem 
includes the examination and development of the theory of the organiza- 
tion and management of social change. Not only are such investigations 
desirable and possible, they are necessary. As we shall see, the very 
structure of a project undergoing an evaluative study may preclude a 
meaningful evaluation, if that structure is not explicitly considered. 

We will consider first the organizational model for the classical 
experimental design, and find it to be the rational or synoptic model. 
When we turn to large-scale research, we will find that this rational 
organizational model is inappropriate. 

Since science is a collaborative and cumulative human activity, it 
is a social activity. It has been long recognized that the relationship 
among scientists constitutes a social system, and recently great concern 
has been exhibited for instance by the American Psychological Associa- 
tion, for the specific nature of this social organization. 

There is, however, a second aspect of the social nature of scien- 
tific activity which has only recently received systematic attention. 

It is now clear that between the scientist and his subject matter exists 
a social relationship as critical to the scientific endeavor as that 
between scientists. It is the social relation of investigator to sub- 
ject matter to which we will attend. 

The traditional view of the social system of investigation and 
subject matter was that of the rational or synoptic model. We will 



briefly summarize the suppositions of this model here. A more complete 
discussion is available in our AERA paper this year. The investigator 
is assumed to have complete knowledge about the subject matter, complete 
control over all variables, and perfect foresight as to the consequences 
of any act. In the literature the assumption prevailed that all parti- 
cipants maximized their individual utilities and, hence, adhered to 
maximizing behaviors. 

Such suppositions about the relationship of investigator to sub- 
ject matter can be maintained in the physical sciences because inputs 
and processes are sufficiently inert and uniform to permit both gener- 
alizability and causal understanding. Consider, for instance, a typical 
physics experiment. 

The research question is, What is the relation between force and 
change in velocity? To ascertain the answer, an experiment is under- 
taken. The inputs include a billiard ball on a glass surface, the 
treatment consists of a constant force applied by a spring loop, and 
the observation is made by flash photography at 10/24 of a second. 

Assurances of control, complete foresight, omniscience, etc. are 
built into this force and velocity experiment. Questions of control 

i 

are dealt with by using standardized instruments. The physical science 
researcher knows that a specified photoflash camera will take one frame 
every 10/24 second; further that the billiard ball will be virtually 
frictionless, certified so by previous experiments, and that spring 
loops respond to given forces in predictable ways. Complete foresight 
is assured by the predictive powers of the experimental results. Know- 
ing the appropriate formula, the researcher derives the solution 14 
cm/sec 2 . He thus generalizes for all time that under the influence of 
a constant force the velocity changes in direct proportion to the time 
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the force acts. Finally, the omniscience, which warrants the supposi- 
tion that his generalization is valid, is due to his ability, so to 
speak, to close the laboratory door. Indeed, he doesn’t even need to 
calculate an F-ratio, his predictive accurace is so good. 

The nature of the subject matter under investigation, hence the 
relationship between researcher and subject, changes when these controls 
and optimal conditions are relaxed. This is particularly the case when 
the subject is a human rather than inert matter. Yet, researchers in 
the social context often assume (at least implicitly) that controls in- 
herent in the physical science situation also obtain with human subjects. 

Economists have long recognized the shortcomings of the rational 
model. Hence, they, the developers of this model, have also noted its 
inappropriateness for wholesale use in social situations . We will sketch 
these shortcomings here. They are three in number. The first can be 
called the personal or computational shortcoming. An individual simply 
lacks the abilities, for instance of data processing, that the rational 
model requires. He is a finite automaton, and it has been proved by 
Gerald Kramer that this type of data processing or calculating mechanism 
simply cannot handle the infinitely large quantities of inputs from the 
environment, which* the classical model supposes. It is not possible in 
the social situation to close the laboratory door, if for no other rea- 
son, because the subjects have a history which is germane to their pres- 
ent behavior. One way to deal with this shortcoming is to introduce 
random assignment into the research project. Thereby the number of 
factors which the research manager must consider are reduced, hopefully 
to a manageable size. But now the system under consideration is not 
represented by a rational model, but rather a stochastic model. 
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A second sort of shortcoming may be called the social psychological * 
Individuals do noc live in the cultural vacuum of Robinson Crusoe as the 
classical model supposes. In the case of the physics experiment, there 
is r.o social interaction of subject matter and investigator; the subject 
matter is utterly inert. The social reality of human subjects is rather 
as von Neumann and Morgenstern point out, one of strategic indepe mce. 
The investigator interacts, strategically, with the subjects. This fact 
radically modifies the nature of the system under study, and the rational 
model is simply inapplicable. The relevant model becomes that of the 
theory of games of strategy. 

The third shortcoming of the rational model is o rganizational . It 
is supposed that there is no difference in operating characteristics of 
a myriad of differing organizational patterns. It is to the inapplica- 
bility of this assumption that we turn. 

In spite of the recognition of the myriad of social mechanisms 
which constrain rationality, numerous evaluation studies assume a priori 
that the social system of the research activity has the characteristics 
of the rational model of organizations. For example, most ESEA Title I 
evaluative reports are based either on the one-shot case study type of 
design or else the analysis of covariance design. These designs assume 
a stable organizational framework in which the program operates. 

For instance-., the Westinghouse-C*hi r , I'nivsrsity study of Headstart 
utilized the covariance analysis of variance design for that nationwide 
study. It was assumed, by virtue of the design employed, that the organ- 
ization of the Headstart program was classical; that is that each class- 
room of the Headstart installation was directly related to the installa- 
tion^ central management. There was no problem of organization for 
these researchers because it was supposed that the rational model is 
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realized. As Edward McDill and his colleagues noted, these suppositions 
are false. This point needs no further emphasis. 



The ramifications of such suppositions for the acceptance of the 
null hypothesis concerning the performance of Headstart versus non-Head- 
start children does warrant emphasis. For on the basis of just such an 
analysis, it was judged that the Headstart children performed no better 
than their peers who had not attended Headstart. 

Since all the possible sources of variation, such as distortion of 
information, etc., are eliminated in the classical model by an assumption 
such as Krathwohl’s ’’typicality of situations," we can suppose that for 
the purposes of the Westinghouse-Ohio University study, the global and 
local performance indices coincide. The presumption is that the system 
is optimized in global and local terms. 

Let us now introduce some complications to our model, in the inter- 
est of realism. The global performance index consists of the local 
performance indices combined with the indices of the losses due to organ- 
izational structure. We will assume that there are losses in performance 
incurred by the biases and noise introduced by communication. 

It follows that as a message comes down through an organization, 
level to level, from say the principal to the classroom teacher, it will 
suffer communication loss at each level. Thus it becomes impossible to 
insure that what happens at the local level conforms to what is expected 
to happen from the global level. As a timely example, the New York 
Times related in March of this year how a General named William Peers 
completed an investigation of the My Lai incident in Viet Nam. Peers 
estimated that 150 to 200 noncombatant civilians were killed. As reports 
of the incident travelled up the organization, from company level, to 
brigade, to the division level, the numbers decreased, until the reports 




- 5 - 



were twenty to thirty killed. This is a good example of communication 
loss in a heirarchical structure, conforming to what one would expect 
in a realistic model of organizations such as that of Anthony Downs. 

Thus we must expect that the mere existence of organization will 
be a relevant factor in large-scale systems such as the national Head- 
start program. 

This evidence means that the researcher cannot argue that the Head- 
start system was optimized at the local level. We seek, however, to 
prove the stronger contention that Headstart was not homogeneous at the 
local level, as the rational model supposes and as the covariance model 
assumes. 

If the number of levels between, say, principal and classroom 
teacher varies, then the amount of loss due to organization or commun- 
ications will vary from classroom to classroom. We will call such an 
organizational pattern idiosyncratic. By a theorem due to the Polish 
mathematician Kulikowski, it can be shown that the project manager must 
be more stringent in resource allocations to the classrooms with the 
greater number of levels in ihe idiosyncratic heirarchy. Conversely, 
the researcher should expect a lower level of output in the classroom 
with the lesser number of levels. 

Where the organizational structure is symmetrical, the manager must 
not distinguish between classrooms in his resource allocations. It 
goes without saying that the typical large-scale research enterprise 
has an organizational structure more idiosyncratic than the symmetric 
structure. Perfect heirarchies are the exception rather than the rule. 
It follows that organizational structure of a complex research enter- 
prise introduces variance into the criterion measure. If the program 
makes equal resource allocations per classroom, then outputs per 



classroom will differ in a systematic fashion. On the other hand, to 
guarantee equality of output, say to meet a mandate for equality of 
educational opportunity, the program manager must vary inputs in a 
systematic fashion. But the latter is a prima facie denial of the 
typicality of situations. 

When we examine the organizational structure which characterized 
Headstart, we will find that the structure was idiosyncratic. Yet the 

research design employed supposed, as we have noted, that the structure 

\ 

was symmetric. 

Within the classroom, the organization of the Headstart project 
was reasonably uniform across the nation. For instance, the Department 
of Health, Education and Welfare, in 1967, reported that in the full- 
year program, 23% of the staff nationwide were teachers and 30% were 
teacher’s aides. Thus there was roughly a pairing of teacher and aide. 

Especially interesting, however, was the structure of the Get Set 
program in Philadelphia. This large city example of Headstart had 
about ten Supervisors, each of which had charge of about ten Centers. 

The organizational pattern here was strictly symmetrical. 

Within each Center in Philadelphia were various numbers of teachers, 
ranging from a couple to 9 or 10. E^ch teacher was assigned one teach- 
er’s aide. One of the teachers was designated Lead Teacher and chan- 
neled all messages from any source up to the Supervisor in charge of 
that Center, and relayed the Supervisor’s comments down to the teachers 
and aides. The lead teacher also was assigned one aide, and had a 
classroom of her own. This patter was thought necessary because of the 
large number of teachers, classrooms, etc. Hence we find in Philadel- 
phia that pattern of organization which we have labelled idiosyncratic. 
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In one classroom per Center in Philadelphia, the classroom of the 
lead teacher, there was less information and communication loss than in 
the other classrooms. Thus by Kulikowski's Second Theorem, the research- 
er would expect a lower level of performance exhibited by the students 
in the lead teacher’s room than in the other classrooms. 

As we’ve noted, there is evidence of similar organizational varia- 
tion elsewhere in the nation. Thus we might suppose that this expecta- 
tion would be held by any researcher engaged in global and national 
evaluative studies. Yet the Coleman study of Equality of Educational 
Opportunity, the Westinghouse-Ohio University study, etc., which are 
examples of recent national studies, give no indication of this concern. 

In conclusion, we would like to re-emphasize that in large-scale 
research and evaluation endeavors, the project’s organizational structure 
itself becomes a source of variance in the research findings. Before 
the researcher publishes his findings of no significant differences, he 
would do well to consider the effect of this variance on the denominator 
of an F-ratio. Certainly more accurate data than our illustrative cases 
are needed to ascertain the effect of structural variation. Such accur- 
ate data does not at present exist. 

We are not at all saying that large-scale research projects are 
undesirable, nor are we saying in this age of computers that they are 
unfeasible. As Richard Light has recently argued in the Harvard Educa- 
tional Review , such projects are feasible if properly conceptualized. 

We merely say that they must be better conceptualized than they have 
been to date. Mr. Lundin's paper will discuss an attempt to better con- 
ceptualize the problems of national programs and their evaluation. 
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